MDEV-21879 GROUP_CONCAT(DISTINCT ORDER BY) is wrong when Unique spills - #5574
MDEV-21879 GROUP_CONCAT(DISTINCT ORDER BY) is wrong when Unique spills#5574arcivanov wants to merge 1 commit into
Conversation
`Item_func_group_concat::add()` decided whether a row was a duplicate
by checking whether `Unique::elements_in_tree()` had grown after
`unique_add()`:
uint count= unique_filter->elements_in_tree();
unique_filter->unique_add(get_record_pointer());
if (count == unique_filter->elements_in_tree())
row_eligible= FALSE;
`Unique` flushes its whole in-memory tree to disk when it runs out of
memory, and `elements_in_tree()` only counts what is still in memory.
After the first flush the test says nothing about the rows that were
already spilled.
**MDEV-11563** made this harmless for `GROUP_CONCAT(DISTINCT x)` by
building the result in `val_str()` from `unique_filter->walk()`, which
merges the spilled parts back in. It left the `ORDER BY` case alone.
There the result comes from the sort tree, which `add()` fills gated by
`row_eligible`, so the defect is still fully live.
Both directions of the failure are reachable, depending on how often
the filter flushes relative to the insert:
1. Duplicates reach the result. 100 rows holding 50 distinct values
give all 100 values back.
2. Rows are lost. 30 distinct rows of 2000 bytes give one value back.
`JSON_ARRAYAGG(DISTINCT x ORDER BY y)` fails in the same way.
Fixed by not filling the sort tree from `add()` when `DISTINCT` is
used. `val_str()` now walks the merged `unique_filter` into the sort
tree and then walks the sort tree, so the rows are sorted after the
duplicate filtering is complete instead of during it.
`Unique::walk()` merges everything it flushed, so the sort tree can be
handed more rows than fit in memory. `insert_to_order_tree()` repacks
it on the same memory budget `add()` used, and a walk that runs out of
memory sets `result_cut`, so the user gets a cut value warning rather
than a silently short result.
**Behaviour change.** `ORDER BY` does not order rows that tie on the
ordering expression, and which of them comes first changes here. It
used to follow the order the rows were read in; it now follows the
order the duplicate filter keeps them in. Unlike the old order, the new
one depends on neither the memory available nor the physical row order.
`main.gconcat_distinct_spill` checks that, and `main.func_gconcat`
records one such tie.
gkodinov
left a comment
There was a problem hiding this comment.
Thank you for your contribution! This is a preliminary review.
LGTM. One small cleanup proposed.
Please stand by for the final review.
| 00,01,10,11,31 | ||
| select group_concat(distinct a, c order by a) from t1; | ||
| group_concat(distinct a, c order by a) | ||
| 00,01,11,10,31 |
There was a problem hiding this comment.
I'd try to stabilize this test instead of re-recording the new undeterministic order.
There was a problem hiding this comment.
it is deterministic, it's just that determinism has changed.
There was a problem hiding this comment.
See Behavior Change section please
|
BTW, any specific reason this is not based on 12.3 (the lowest affected version according to Jira)? |
Yes, because it depends on other commits that are specifically in bb-blob-main-monty. I'll discuss this with @montywi if/when he is available. There may be further work around GROUP_CONCAT as well (e.g. mem -> HEAP -> Aria spillover mechanisms to eliminate the avoidable truncation of the results). |
GROUP_CONCAT(DISTINCT x ORDER BY y)andJSON_ARRAYAGG(DISTINCT x ORDER BY y)return a wrong answer once the duplicate filter runs out of memory.
The defect
Item_func_group_concat::add()decides whether a row is a duplicate by checkingwhether
Unique::elements_in_tree()grew afterunique_add().Uniqueflushesits whole in-memory tree to disk when it runs out of memory, and
elements_in_tree()only counts what is still in memory, so after the firstflush that test says nothing about the rows already spilled.
MDEV-11563 made this harmless for
GROUP_CONCAT(DISTINCT x)by building theresult in
val_str()fromunique_filter->walk(), which merges the spilledparts back in. It left the
ORDER BYcase alone, where the result comes from thesort tree that
add()fills gated by the broken test.Both directions of the failure are reachable, depending on how often the filter
flushes relative to the insert:
100 values back.
The fix
add()no longer fills the sort tree whenDISTINCTis used.val_str()walksthe merged
unique_filterinto the sort tree and then walks the sort tree, so therows are sorted after duplicate filtering is complete instead of during it.
Unique::walk()merges everything it flushed, so the sort tree can be handed morerows than fit in memory.
insert_to_order_tree()repacks it on the same memorybudget
add()used, and a walk that runs out of memory setsresult_cutso theuser gets a cut value warning rather than a silently short result.
This targets
bb-blob-main-montyrather thanmainbecause it relies on37077ccef15"Limit the memory used by GROUP_CONCAT() with ORDER BY". Withoutthat commit's
tree->allocatedbound and itsst.oomcorrection torepack_tree(), pouring a spilledUniqueinto the sort tree trades a wronganswer for an out-of-memory error.
Behaviour change
ORDER BYdoes not order rows that tie on the ordering expression, and which ofthem comes first changes here. It used to follow the order the rows were read in;
it now follows the order the duplicate filter keeps them in. Unlike the old order,
the new one depends on neither the memory available nor the physical row order.
main.func_gconcatrecords one such tie and is re-recorded accordingly.Testing
New
main.gconcat_distinct_spillcovers three things, and each assertion wasconfirmed to fail before the fix and pass after, with the test unchanged between
the two runs:
for
GROUP_CONCATandJSON_ARRAYAGG, with and withoutORDER BYrepack_tree()cuts the group, whatevercomes back must still be deduplicated, still ordered, and still valid JSON
order
Full
main+heapsuite: 1436/1436 pass.